Repository navigation
Conversation
A trusted key's age counted from when the node first received it, and a
/keys sync kept that time. Pruning keeps only the newest key past the
grace period, so a key the control plane still lists in its rotation grace
period was dropped a day after the node first saw it, even when it signed
the node's own identity. The next start then failed the role check
("biscuit: invalid signature") on a valid identity, and if the control
plane could not be reached at that moment, both recoveries failed too.
- mergeTrustedKeys: every key a /keys answer lists counts as vouched for
at that sync, so pruning only drops keys the control plane stopped
listing, or ones the node could not confirm for longer than the grace
period.
- pruneTrustedKeys: keeps every key with the latest time, since a /keys
answer vouches for all its keys at once and its order says nothing about
which is current, and never drops the key that signed the node's current
identity.
- NewAgentMeshNode: the configured key re-added next to persisted keys gets
a fixed old time (the zero time is not stored and read back as now), so
it cannot outrank the control plane's newer keys after a restart.
- Start returns *StaleIdentityError when the stored identity fails the role
check and neither recovery renews it. It unwraps to the role, refresh and
recovery errors, so callers can tell a refusal (*RefreshError with its
status) from a control plane that never answered.
There was a problem hiding this comment.
Code Review
This pull request updates the trusted key management and pruning logic. It simplifies key merging by updating the received time for all active keys, introduces a structured StaleIdentityError for better error unwrapping, and ensures that the key signing the node's own identity is preserved during pruning. Additionally, it prevents configured fallback keys from outranking newer keys. The review feedback identifies style guide violations in the new tests where errors from ed25519.GenerateKey are ignored; these should be resolved by using the mustGenerateKey(t) helper.
Use mustGenerateKey instead of ignoring ed25519.GenerateKey's error, and report a failed store close (errcheck).
aojea
left a comment
There was a problem hiding this comment.
Thanks for the report and the fix. The bug is real. On main, with the real functions: enrollment gives the node the current key, the first /keys sync adds the retiring key with a later time, and after 24 hours pruning keeps only the retiring key.
The fix for #916 is smaller than this PR. Please reduce the PR to that fix.
Keep
- The sync gives each key in the
/keysanswer the time of the sync. - Pruning keeps all the keys that have the latest time, not only the first one.
Change
- Remove
mergeTrustedKeys. It does not merge now, andexistinghas no effect. Put the loop insyncTrustedKeys. - Add a log line for each key that pruning drops, so that the node log shows the cause the next time.
- Correct the text that still has the old meaning ("first learned"):
receive_timeofTrustedSigningKeyinapi/agentmesh.proto. Then runmake proto, andnpm ciinsdk/jsfollowed by./hack/gen-sdk-proto.sh.- The
--key-grace-periodrow insite/content/docs/reference/agentmesh-node.md.
- Tests: one subtest for the sequence of the issue (sync, then prune) and one row in
TestPruneTrustedKeys.
Remove from this PR
- The
keepparameter andsignedOwnIdentity. neverVouched.StaleIdentityError.
The inline comments give the reason and the code for each item. With these changes applied on main, go test -race ./internal/node/ ./sdk/meshnet/ and make lint pass, and the two new tests fail without the fix.
Proposed text for item 5:
// When the control plane last vouched for the key: the last /keys answer
// that listed it, or the enrollment or rotation event that announced it. A
// member that prunes drops a key a grace period after this, unless no key
// has a later time; unset means unknown and is read as now.
google.protobuf.Timestamp receive_time = 2;| `--key-grace-period` | `24h` | How long the node keeps a control plane key after the control plane last confirmed it in a `/keys` answer or a rotation event. The keys of the latest confirmation always stay. A key that `/keys` no longer lists is removed at the next sync. || // mergeTrustedKeys replaces the stored trust set with the authoritative set | ||
| // from /keys, preserving ReceivedAt for keys already known so local grace | ||
| // pruning keeps working across syncs. | ||
| // from /keys. The control plane lists every key that is still valid, its | ||
| // current key and any in their rotation grace period, so each listed key's | ||
| // ReceivedAt becomes now: local pruning then ages a key from the last time | ||
| // the control plane vouched for it, and never drops one it still lists. | ||
| func mergeTrustedKeys(existing []TrustedKey, fetched []ed25519.PublicKey, now time.Time) []TrustedKey { | ||
| merged := make([]TrustedKey, 0, len(fetched)) | ||
| for _, key := range fetched { | ||
| tk := TrustedKey{Key: key, ReceivedAt: now} | ||
| for _, old := range existing { | ||
| if bytes.Equal(old.Key, key) { | ||
| tk.ReceivedAt = old.ReceivedAt | ||
| break | ||
| } | ||
| } | ||
| merged = append(merged, tk) | ||
| merged = append(merged, TrustedKey{Key: key, ReceivedAt: now}) | ||
| } | ||
| return merged | ||
| } |
There was a problem hiding this comment.
existing has no effect now, and the function does not merge. Please delete this function and TestMergeTrustedKeys, and give the keys their time in syncTrustedKeys:
// The answer lists every key the control plane still accepts, so it
// vouches for each of them now: pruning ages a key from the last answer
// that listed it, not from the first time the node saw it.
now := time.Now()
synced := make([]TrustedKey, 0, len(keys))
for _, key := range keys {
synced = append(synced, TrustedKey{Key: key, ReceivedAt: now})
}syncTrustedKeys still reads existing, to verify the answer.
| // StaleIdentityError is Start's error when the stored identity fails the | ||
| // role check against every trusted key and neither recovery renewed it. It | ||
| // unwraps to the role check's error and to both recovery errors, so a caller | ||
| // can tell a control plane that refused the identity (a *RefreshError with | ||
| // its status) from one that never answered. | ||
| type StaleIdentityError struct { | ||
| Role string | ||
| RoleErr error | ||
| RefreshErr error | ||
| RecoveryErr error | ||
| } | ||
|
|
||
| func (e *StaleIdentityError) Error() string { | ||
| return fmt.Sprintf("loaded identity fails role requirement %q (autonomous refresh: %v; refresh-token recovery: %v): %v", e.Role, e.RefreshErr, e.RecoveryErr, e.RoleErr) | ||
| } | ||
|
|
||
| func (e *StaleIdentityError) Unwrap() []error { | ||
| var errs []error | ||
| for _, err := range []error{e.RoleErr, e.RefreshErr, e.RecoveryErr} { | ||
| if err != nil { | ||
| errs = append(errs, err) | ||
| } | ||
| } | ||
| return errs | ||
| } |
There was a problem hiding this comment.
#916 does not need this type, and nothing in this PR reads it. Please remove it here, together with its test, and send it in a separate PR with the code that uses it.
| // The configured key is only as fresh as the enrollment or /info | ||
| // answer that stored it. Next to persisted keys it must not count as | ||
| // just vouched for, or it would outrank the control plane's newer | ||
| // keys when pruning picks the newest; alone, it is all there is. | ||
| vouched := neverVouched | ||
| if len(trustedKeys) == 0 { | ||
| vouched = time.Now() | ||
| } | ||
| trustedKeys = append(trustedKeys, TrustedKey{Key: cfg.ControlPlanePubKey, ReceivedAt: vouched}) |
There was a problem hiding this comment.
This is a different problem from #916. LoadMeshConfig returns the first stored key, so the node comes here only when --control-plane-public-key gives a key that is not in the stored set. The change also writes the epoch as a special value into the persisted MemberCredential, which the SDKs load too.
Please remove it from this PR, with neverVouched and its test. If you can reproduce the problem, open an issue with the steps.
| // pruneTrustedKeys drops keys the control plane has not vouched for within | ||
| // gracePeriod, as when it stopped listing them or the node could not reach | ||
| // it, but always keeps the most recently vouched-for ones: the current | ||
| // signing key has no successor until a rotation event arrives, so ageing it | ||
| // out would empty the trust set and leave the node unable to verify any peer | ||
| // or router. A /keys answer vouches for all its keys at once and says | ||
| // nothing about which is current, so every key with the latest time is kept, | ||
| // never one of them by its place in the list. It also keeps every key keep reports, which is the key that | ||
| // signed the node's own identity: the control plane lists all its valid keys | ||
| // alike, so the newest vouched-for key need not be the one a refresh signed | ||
| // with, and dropping that one after a long time offline leaves a valid | ||
| // identity that fails the check at the next start, when only the control | ||
| // plane can renew it. | ||
| func pruneTrustedKeys(keys []TrustedKey, now time.Time, gracePeriod time.Duration, keep func(ed25519.PublicKey) bool) []TrustedKey { |
There was a problem hiding this comment.
Please remove the keep parameter and signedOwnIdentity:
- Node prunes its identity’s signing key while /keys still advertises it #916 does not need them. When every listed key has the time of the sync, the sequence of the issue keeps both keys with
keep == nil. - They have no time limit. The signer of an identity that expired long ago stays in the trust set while that identity is in the store.
The loop below stays as it is, without keep. The comment becomes shorter, for example:
// pruneTrustedKeys drops the keys the control plane last vouched for more
// than gracePeriod ago, but always keeps the ones with the latest time: a
// node that cannot reach the control plane must not age out its whole trust
// set and lose the means to verify any peer or router. A /keys answer vouches
// for all its keys at once and its order does not say which is current, so
// the latest time keeps all of them, not the first one in the list.
func pruneTrustedKeys(keys []TrustedKey, now time.Time, gracePeriod time.Duration) []TrustedKey {In startKeyPruning, log each key that pruning drops:
n.keysMu.Lock()
before := n.trustedKeys
pruned := pruneTrustedKeys(before, time.Now(), gracePeriod)
changed := len(pruned) != len(before)
n.trustedKeys = pruned
snapshot := append([]TrustedKey(nil), pruned...)
n.keysMu.Unlock()
for _, tk := range before {
if !containsTrustedKey(pruned, tk.Key) {
logger.Infow("[KeyPruning] dropped control plane key", "key", fmt.Sprintf("%x", tk.Key), "last_vouched", tk.ReceivedAt.UTC().Format(time.RFC3339), "grace_period", gracePeriod.String())
}
}| // TestKeyPruningKeepsTheIdentitySigner covers a node whose identity was | ||
| // signed by a key it learned before the newest one it holds, as when the | ||
| // control plane's newest key reached it earlier or its stored key was added | ||
| // again at a restart: pruning must not drop the signer, or the next start | ||
| // finds a valid identity that no trusted key verifies. | ||
| func TestKeyPruningKeepsTheIdentitySigner(t *testing.T) { |
There was a problem hiding this comment.
For the reduced change, two tests are sufficient. Both fail on main.
- A subtest in
TestSyncTrustedKeysfor the sequence of the issue:
// The node got the current key at enrollment and the retiring key from a
// later sync. While /keys lists both, pruning must keep both, whatever
// the time since the node first saw each (#916). A key that /keys no
// longer lists is dropped at the sync.
t.Run("a key /keys still lists outlives the grace period", func(t *testing.T) {
expiredPub, _ := mustGenerateKey(t)
srv := keysServer(t, []ed25519.PublicKey{retiredPub, currentPub}, []ed25519.PrivateKey{retiredPriv, currentPriv})
defer srv.Close()
n := newNode(t)
enrolled := time.Now().Add(-25 * time.Hour)
n.trustedKeys = []TrustedKey{{Key: currentPub, ReceivedAt: enrolled}, {Key: expiredPub, ReceivedAt: enrolled}}
if err := n.syncTrustedKeys(context.Background(), srv.URL); err != nil {
t.Fatalf("syncTrustedKeys: %v", err)
}
if containsTrustedKey(n.trustedKeys, expiredPub) {
t.Error("a key /keys does not list stayed in the trust set")
}
got := pruneTrustedKeys(n.trustedKeys, time.Now(), 24*time.Hour)
if len(got) != 2 || !containsTrustedKey(got, retiredPub) || !containsTrustedKey(got, currentPub) {
t.Errorf("pruning kept %d keys, want the retired and the current key that /keys lists", len(got))
}
})- A row in
TestPruneTrustedKeys. Add it at the end, because the table setstests[3].wantby index:
sameSync := k(48 * time.Hour) {"every key of the latest sync survives", []TrustedKey{stale, staler, sameSync}, []TrustedKey{stale, sameSync}},Please remove TestKeyPruningKeepsTheIdentitySigner, TestKeyPruningKeepsEveryLatestKey, TestFallbackKeyStaysOldThroughTheStore and TestStaleIdentityErrorUnwraps together with the code that they test.
A trusted key's age counted from when the node first received it, and a /keys sync kept that time. Pruning keeps only the newest key past the grace period, so a key the control plane still lists in its rotation grace period was dropped a day after the node first saw it, even when it signed the node's own identity. The next start then failed the role check ("biscuit: invalid signature") on a valid identity, and if the control plane could not be reached at that moment, both recoveries failed too.
Fixes: #916